Visualizing what convnets learn

In [1]:
import keras
keras.__version__
Using TensorFlow backend.
Out[1]:
'2.2.5'
In [1]:
!sudo pip install pillow
WARNING: The directory '/home/jovyan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/home/jovyan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting pillow
  Downloading https://files.pythonhosted.org/packages/14/41/db6dec65ddbc176a59b89485e8cc136a433ed9c6397b6bfe2cd38412051e/Pillow-6.1.0-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)
     |████████████████████████████████| 2.1MB 3.2MB/s eta 0:00:01
Installing collected packages: pillow
Successfully installed pillow-6.1.0
WARNING: You are using pip version 19.1.1, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

There are three methods of visualization: Visualizing intermediate convnet outputs, Visualizing convnets filters and Visualizing heatmaps of class activation in an image. For the first method, we will use the small convnet used in 5.2 section. For the next two methods, we will use the VGG16 model. So first I need to generate the model in 5.2.

Generate the model in 5.2

before get to the visualization, we should first run the model in 5.2. The model will have a stack of alternated Conv2D (with relu activation) and MaxPooling2D layers:

In [2]:
import os, shutil
In [3]:
from keras import layers
from keras import models

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu',
                        input_shape=(150, 150, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
WARNING: Logging before flag parsing goes to stderr.
W0914 20:16:01.891050 139711336847168 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

W0914 20:16:01.914268 139711336847168 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0914 20:16:01.917285 139711336847168 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

W0914 20:16:01.934117 139711336847168 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4267: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

In [4]:
model.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 148, 148, 32)      896       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 74, 74, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 72, 72, 64)        18496     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 36, 36, 64)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 34, 34, 128)       73856     
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 17, 17, 128)       0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 15, 15, 128)       147584    
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 7, 7, 128)         0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 6272)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 512)               3211776   
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 513       
=================================================================
Total params: 3,453,121
Trainable params: 3,453,121
Non-trainable params: 0
_________________________________________________________________
In [5]:
from keras import optimizers

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=1e-4),
              metrics=['acc'])
W0914 20:16:06.413683 139711336847168 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

W0914 20:16:06.420619 139711336847168 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3657: The name tf.log is deprecated. Please use tf.math.log instead.

W0914 20:16:06.426119 139711336847168 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where

Data preprocessing: using ImageDataGenerator in keras.preprocessing.image to turn image files on disk into batches of pre-processed tensors:

In [6]:
from keras.preprocessing.image import ImageDataGenerator

base_dir = 'Fruits_Data5'
train_dir = os.path.join(base_dir, 'train')
validation_dir = os.path.join(base_dir, 'validation')
test_dir = os.path.join(base_dir, 'test')

# All images will be rescaled by 1./255
train_datagen = ImageDataGenerator(rescale=1./255)
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        # This is the target directory
        train_dir,
        # All images will be resized to 150x150
        target_size=(150, 150),
        batch_size=20,
        # Since we use binary_crossentropy loss, we need binary labels
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=20,
        class_mode='binary')
Found 60 images belonging to 2 classes.
Found 30 images belonging to 2 classes.
In [7]:
for data_batch, labels_batch in train_generator:
    print('data batch shape:', data_batch.shape)
    print('labels batch shape:', labels_batch.shape)
    break
data batch shape: (20, 150, 150, 3)
labels batch shape: (20,)
/usr/local/lib/python3.6/dist-packages/PIL/Image.py:993: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
  "Palette images with Transparency expressed in bytes should be "

Then fit the model to the data using the generator:

In [8]:
history = model.fit_generator(
      train_generator,
      steps_per_epoch=4,
      epochs=30,
      validation_data=validation_generator,
      validation_steps=2)
W0914 20:16:16.633130 139711336847168 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

Epoch 1/30
/usr/local/lib/python3.6/dist-packages/PIL/Image.py:993: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
  "Palette images with Transparency expressed in bytes should be "
4/4 [==============================] - 3s 731ms/step - loss: 0.7104 - acc: 0.7000 - val_loss: 0.6429 - val_acc: 0.6333
Epoch 2/30
4/4 [==============================] - 2s 450ms/step - loss: 0.5719 - acc: 0.7625 - val_loss: 0.5246 - val_acc: 0.7333
Epoch 3/30
4/4 [==============================] - 2s 537ms/step - loss: 0.4202 - acc: 0.8500 - val_loss: 0.4954 - val_acc: 0.6000
Epoch 4/30
4/4 [==============================] - 2s 464ms/step - loss: 0.3615 - acc: 0.8750 - val_loss: 0.3440 - val_acc: 0.9000
Epoch 5/30
4/4 [==============================] - 2s 547ms/step - loss: 0.2519 - acc: 0.9000 - val_loss: 0.3832 - val_acc: 0.8667
Epoch 6/30
4/4 [==============================] - 2s 534ms/step - loss: 0.2392 - acc: 0.9500 - val_loss: 0.2159 - val_acc: 0.9333
Epoch 7/30
4/4 [==============================] - 2s 524ms/step - loss: 0.1569 - acc: 0.9625 - val_loss: 0.1866 - val_acc: 0.9333
Epoch 8/30
4/4 [==============================] - 2s 507ms/step - loss: 0.1101 - acc: 0.9625 - val_loss: 0.1847 - val_acc: 0.9333
Epoch 9/30
4/4 [==============================] - 2s 502ms/step - loss: 0.1184 - acc: 0.9750 - val_loss: 0.1750 - val_acc: 0.9333
Epoch 10/30
4/4 [==============================] - 2s 499ms/step - loss: 0.0641 - acc: 0.9875 - val_loss: 0.1374 - val_acc: 0.9333
Epoch 11/30
4/4 [==============================] - 2s 481ms/step - loss: 0.0622 - acc: 1.0000 - val_loss: 0.1657 - val_acc: 0.9000
Epoch 12/30
4/4 [==============================] - 2s 487ms/step - loss: 0.0766 - acc: 0.9625 - val_loss: 0.1125 - val_acc: 0.9333
Epoch 13/30
4/4 [==============================] - 2s 549ms/step - loss: 0.0499 - acc: 1.0000 - val_loss: 0.1596 - val_acc: 0.9333
Epoch 14/30
4/4 [==============================] - 4s 1s/step - loss: 0.0258 - acc: 1.0000 - val_loss: 0.1292 - val_acc: 0.9333
Epoch 15/30
4/4 [==============================] - 4s 1s/step - loss: 0.1158 - acc: 0.9500 - val_loss: 0.3117 - val_acc: 0.8667
Epoch 16/30
4/4 [==============================] - 4s 1000ms/step - loss: 0.0372 - acc: 1.0000 - val_loss: 0.0771 - val_acc: 1.0000
Epoch 17/30
4/4 [==============================] - 4s 1s/step - loss: 0.0192 - acc: 1.0000 - val_loss: 0.0851 - val_acc: 0.9667
Epoch 18/30
4/4 [==============================] - 4s 1s/step - loss: 0.0158 - acc: 1.0000 - val_loss: 0.0730 - val_acc: 1.0000
Epoch 19/30
4/4 [==============================] - 4s 1s/step - loss: 0.0141 - acc: 1.0000 - val_loss: 0.0991 - val_acc: 0.9333
Epoch 20/30
4/4 [==============================] - 4s 1s/step - loss: 0.0162 - acc: 1.0000 - val_loss: 0.0895 - val_acc: 0.9667
Epoch 21/30
4/4 [==============================] - 4s 1s/step - loss: 0.0234 - acc: 1.0000 - val_loss: 0.0470 - val_acc: 1.0000
Epoch 22/30
4/4 [==============================] - 4s 1s/step - loss: 0.0078 - acc: 1.0000 - val_loss: 0.0602 - val_acc: 0.9667
Epoch 23/30
4/4 [==============================] - 4s 1000ms/step - loss: 0.0069 - acc: 1.0000 - val_loss: 0.0586 - val_acc: 0.9667
Epoch 24/30
4/4 [==============================] - 4s 1s/step - loss: 0.0068 - acc: 1.0000 - val_loss: 0.0651 - val_acc: 0.9667
Epoch 25/30
4/4 [==============================] - 4s 1s/step - loss: 0.0049 - acc: 1.0000 - val_loss: 0.0497 - val_acc: 1.0000
Epoch 26/30
4/4 [==============================] - 4s 1s/step - loss: 0.0086 - acc: 1.0000 - val_loss: 0.4772 - val_acc: 0.8333
Epoch 27/30
4/4 [==============================] - 4s 1s/step - loss: 0.1844 - acc: 0.9000 - val_loss: 0.0299 - val_acc: 1.0000
Epoch 28/30
4/4 [==============================] - 4s 1s/step - loss: 0.0029 - acc: 1.0000 - val_loss: 0.0286 - val_acc: 1.0000
Epoch 29/30
4/4 [==============================] - 4s 998ms/step - loss: 0.0037 - acc: 1.0000 - val_loss: 0.0298 - val_acc: 1.0000
Epoch 30/30
4/4 [==============================] - 4s 1s/step - loss: 0.0025 - acc: 1.0000 - val_loss: 0.0294 - val_acc: 1.0000

Save the model and visualize the loss and accuracy:

In [9]:
model.save('fruits_classification_1.h5')
In [11]:
import matplotlib.pyplot as plt

acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()

Using data augmentation:

In [12]:
datagen = ImageDataGenerator(
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')
In [13]:
train_apple_dir = os.path.join(train_dir, 'apple')
train_orange_dir = os.path.join(train_dir, 'orange')
validation_apple_dir = os.path.join(validation_dir, 'apple')
validation_orange_dir = os.path.join(validation_dir, 'orange')
test_apple_dir = os.path.join(test_dir, 'apple')
test_orange_dir = os.path.join(test_dir, 'orange')
In [14]:
# This is module with image preprocessing utilities
from keras.preprocessing import image

fnames = [os.path.join(train_apple_dir, fname) for fname in os.listdir(train_apple_dir)]

# We pick one image to "augment"
img_path = fnames[3]

# Read the image and resize it
img = image.load_img(img_path, target_size=(150, 150))

# Convert it to a Numpy array with shape (150, 150, 3)
x = image.img_to_array(img)

# Reshape it to (1, 150, 150, 3)
x = x.reshape((1,) + x.shape)

# The .flow() command below generates batches of randomly transformed images.
# It will loop indefinitely, so we need to `break` the loop at some point!
i = 0
for batch in datagen.flow(x, batch_size=1):
    plt.figure(i)
    imgplot = plt.imshow(image.array_to_img(batch[0]))
    i += 1
    if i % 4 == 0:
        break

plt.show()

Add a Dropout layer to our model, right before the densely-connected classifier:

In [15]:
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu',
                        input_shape=(150, 150, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dropout(0.5))
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.RMSprop(lr=1e-4),
              metrics=['acc'])
W0914 20:22:54.316912 139711336847168 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3733: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.

Train the model:

In [16]:
train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=40,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,)

# Note that the validation data should not be augmented!
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        # This is the target directory
        train_dir,
        # All images will be resized to 150x150
        target_size=(150, 150),
        batch_size=32,
        # Since we use binary_crossentropy loss, we need binary labels
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

history = model.fit_generator(
      train_generator,
      steps_per_epoch=4,
      epochs=100,
      validation_data=validation_generator,
      validation_steps=2)
Found 60 images belonging to 2 classes.
Found 30 images belonging to 2 classes.
Epoch 1/100
/usr/local/lib/python3.6/dist-packages/PIL/Image.py:993: UserWarning: Palette images with Transparency expressed in bytes should be converted to RGBA images
  "Palette images with Transparency expressed in bytes should be "
4/4 [==============================] - 8s 2s/step - loss: 0.6954 - acc: 0.5520 - val_loss: 0.6818 - val_acc: 0.5000
Epoch 2/100
4/4 [==============================] - 7s 2s/step - loss: 0.6327 - acc: 0.7245 - val_loss: 0.5836 - val_acc: 0.7000
Epoch 3/100
4/4 [==============================] - 7s 2s/step - loss: 0.5257 - acc: 0.8230 - val_loss: 0.7035 - val_acc: 0.5000
Epoch 4/100
4/4 [==============================] - 6s 1s/step - loss: 0.4665 - acc: 0.7649 - val_loss: 0.3960 - val_acc: 0.9000
Epoch 5/100
4/4 [==============================] - 4s 889ms/step - loss: 0.4276 - acc: 0.8002 - val_loss: 0.3587 - val_acc: 0.9000
Epoch 6/100
4/4 [==============================] - 4s 925ms/step - loss: 0.3630 - acc: 0.8768 - val_loss: 0.2802 - val_acc: 0.9000
Epoch 7/100
4/4 [==============================] - 4s 925ms/step - loss: 0.2193 - acc: 0.9595 - val_loss: 0.1966 - val_acc: 0.9333
Epoch 8/100
4/4 [==============================] - 3s 852ms/step - loss: 0.1983 - acc: 0.9419 - val_loss: 0.2984 - val_acc: 0.8667
Epoch 9/100
4/4 [==============================] - 4s 979ms/step - loss: 0.2941 - acc: 0.8698 - val_loss: 0.1599 - val_acc: 0.9333
Epoch 10/100
4/4 [==============================] - 4s 898ms/step - loss: 0.1401 - acc: 0.9568 - val_loss: 0.1460 - val_acc: 0.9333
Epoch 11/100
4/4 [==============================] - 4s 974ms/step - loss: 0.1563 - acc: 0.9498 - val_loss: 0.2066 - val_acc: 0.9000
Epoch 12/100
4/4 [==============================] - 4s 902ms/step - loss: 0.1423 - acc: 0.9419 - val_loss: 0.2222 - val_acc: 0.8667
Epoch 13/100
4/4 [==============================] - 3s 851ms/step - loss: 0.1576 - acc: 0.9252 - val_loss: 0.3074 - val_acc: 0.8667
Epoch 14/100
4/4 [==============================] - 4s 977ms/step - loss: 0.1684 - acc: 0.9313 - val_loss: 0.1367 - val_acc: 0.9667
Epoch 15/100
4/4 [==============================] - 3s 842ms/step - loss: 0.1291 - acc: 0.9665 - val_loss: 0.1351 - val_acc: 0.9667
Epoch 16/100
4/4 [==============================] - 4s 978ms/step - loss: 0.1401 - acc: 0.9437 - val_loss: 0.2176 - val_acc: 0.9000
Epoch 17/100
4/4 [==============================] - 4s 878ms/step - loss: 0.2113 - acc: 0.8899 - val_loss: 0.1077 - val_acc: 0.9667
Epoch 18/100
4/4 [==============================] - 4s 974ms/step - loss: 0.0658 - acc: 0.9833 - val_loss: 0.0943 - val_acc: 0.9667
Epoch 19/100
4/4 [==============================] - 4s 958ms/step - loss: 0.0519 - acc: 0.9754 - val_loss: 0.0862 - val_acc: 0.9667
Epoch 20/100
4/4 [==============================] - 3s 867ms/step - loss: 0.0723 - acc: 0.9745 - val_loss: 0.0867 - val_acc: 0.9667
Epoch 21/100
4/4 [==============================] - 4s 1s/step - loss: 0.0618 - acc: 0.9745 - val_loss: 0.1018 - val_acc: 0.9667
Epoch 22/100
4/4 [==============================] - 4s 909ms/step - loss: 0.2222 - acc: 0.9331 - val_loss: 0.1395 - val_acc: 0.9667
Epoch 23/100
4/4 [==============================] - 4s 961ms/step - loss: 0.1028 - acc: 0.9745 - val_loss: 0.0822 - val_acc: 0.9667
Epoch 24/100
4/4 [==============================] - 4s 948ms/step - loss: 0.0697 - acc: 0.9736 - val_loss: 0.1369 - val_acc: 0.9333
Epoch 25/100
4/4 [==============================] - 5s 1s/step - loss: 0.0400 - acc: 1.0000 - val_loss: 0.0744 - val_acc: 0.9667
Epoch 26/100
4/4 [==============================] - 6s 2s/step - loss: 0.0559 - acc: 0.9912 - val_loss: 0.0729 - val_acc: 0.9667
Epoch 27/100
4/4 [==============================] - 6s 2s/step - loss: 0.0356 - acc: 0.9912 - val_loss: 0.0696 - val_acc: 0.9667
Epoch 28/100
4/4 [==============================] - 7s 2s/step - loss: 0.0304 - acc: 0.9921 - val_loss: 0.0656 - val_acc: 0.9667
Epoch 29/100
4/4 [==============================] - 7s 2s/step - loss: 0.0673 - acc: 0.9824 - val_loss: 0.4016 - val_acc: 0.9000
Epoch 30/100
4/4 [==============================] - 7s 2s/step - loss: 0.0993 - acc: 0.9358 - val_loss: 0.0592 - val_acc: 0.9667
Epoch 31/100
4/4 [==============================] - 7s 2s/step - loss: 0.0323 - acc: 0.9912 - val_loss: 0.0518 - val_acc: 0.9667
Epoch 32/100
4/4 [==============================] - 7s 2s/step - loss: 0.0232 - acc: 0.9921 - val_loss: 0.1148 - val_acc: 0.9667
Epoch 33/100
4/4 [==============================] - 7s 2s/step - loss: 0.0480 - acc: 0.9762 - val_loss: 0.0440 - val_acc: 1.0000
Epoch 34/100
4/4 [==============================] - 7s 2s/step - loss: 0.0212 - acc: 0.9921 - val_loss: 0.0469 - val_acc: 0.9667
Epoch 35/100
4/4 [==============================] - 7s 2s/step - loss: 0.0299 - acc: 0.9921 - val_loss: 0.1522 - val_acc: 0.9333
Epoch 36/100
4/4 [==============================] - 4s 1s/step - loss: 0.0685 - acc: 0.9674 - val_loss: 0.0410 - val_acc: 1.0000
Epoch 37/100
4/4 [==============================] - 5s 1s/step - loss: 0.0093 - acc: 1.0000 - val_loss: 0.0367 - val_acc: 1.0000
Epoch 38/100
4/4 [==============================] - 6s 2s/step - loss: 0.0809 - acc: 0.9754 - val_loss: 0.1133 - val_acc: 0.9333
Epoch 39/100
4/4 [==============================] - 6s 2s/step - loss: 0.0118 - acc: 1.0000 - val_loss: 0.0349 - val_acc: 1.0000
Epoch 40/100
4/4 [==============================] - 6s 2s/step - loss: 0.0464 - acc: 0.9745 - val_loss: 0.0748 - val_acc: 0.9333
Epoch 41/100
4/4 [==============================] - 7s 2s/step - loss: 0.0256 - acc: 0.9921 - val_loss: 0.0758 - val_acc: 0.9667
Epoch 42/100
4/4 [==============================] - 7s 2s/step - loss: 0.0153 - acc: 1.0000 - val_loss: 0.0678 - val_acc: 0.9333
Epoch 43/100
4/4 [==============================] - 6s 2s/step - loss: 0.0143 - acc: 1.0000 - val_loss: 0.0293 - val_acc: 1.0000
Epoch 44/100
4/4 [==============================] - 4s 1s/step - loss: 0.0054 - acc: 1.0000 - val_loss: 0.0284 - val_acc: 1.0000
Epoch 45/100
4/4 [==============================] - 6s 2s/step - loss: 0.0138 - acc: 1.0000 - val_loss: 0.0242 - val_acc: 1.0000
Epoch 46/100
4/4 [==============================] - 7s 2s/step - loss: 0.0199 - acc: 0.9912 - val_loss: 0.0604 - val_acc: 0.9667
Epoch 47/100
4/4 [==============================] - 6s 2s/step - loss: 0.0510 - acc: 0.9833 - val_loss: 0.0283 - val_acc: 1.0000
Epoch 48/100
4/4 [==============================] - 6s 2s/step - loss: 0.0145 - acc: 0.9912 - val_loss: 0.2210 - val_acc: 0.9333
Epoch 49/100
4/4 [==============================] - 7s 2s/step - loss: 0.0096 - acc: 1.0000 - val_loss: 0.0215 - val_acc: 1.0000
Epoch 50/100
4/4 [==============================] - 6s 2s/step - loss: 0.0100 - acc: 1.0000 - val_loss: 0.0487 - val_acc: 0.9667
Epoch 51/100
4/4 [==============================] - 7s 2s/step - loss: 0.0124 - acc: 1.0000 - val_loss: 0.0450 - val_acc: 0.9667
Epoch 52/100
4/4 [==============================] - 7s 2s/step - loss: 0.0298 - acc: 0.9833 - val_loss: 0.0276 - val_acc: 1.0000
Epoch 53/100
4/4 [==============================] - 7s 2s/step - loss: 0.0242 - acc: 0.9824 - val_loss: 0.0169 - val_acc: 1.0000
Epoch 54/100
4/4 [==============================] - 4s 1s/step - loss: 0.0095 - acc: 1.0000 - val_loss: 0.0150 - val_acc: 1.0000
Epoch 55/100
4/4 [==============================] - 7s 2s/step - loss: 0.0039 - acc: 1.0000 - val_loss: 0.0617 - val_acc: 0.9667
Epoch 56/100
4/4 [==============================] - 6s 2s/step - loss: 0.0049 - acc: 1.0000 - val_loss: 0.0466 - val_acc: 0.9667
Epoch 57/100
4/4 [==============================] - 7s 2s/step - loss: 0.0995 - acc: 0.9498 - val_loss: 0.0151 - val_acc: 1.0000
Epoch 58/100
4/4 [==============================] - 6s 1s/step - loss: 0.0024 - acc: 1.0000 - val_loss: 0.0161 - val_acc: 1.0000
Epoch 59/100
4/4 [==============================] - 5s 1s/step - loss: 0.0031 - acc: 1.0000 - val_loss: 0.0190 - val_acc: 1.0000
Epoch 60/100
4/4 [==============================] - 3s 867ms/step - loss: 0.0040 - acc: 1.0000 - val_loss: 0.0129 - val_acc: 1.0000
Epoch 61/100
4/4 [==============================] - 4s 931ms/step - loss: 0.0016 - acc: 1.0000 - val_loss: 0.0139 - val_acc: 1.0000
Epoch 62/100
4/4 [==============================] - 5s 1s/step - loss: 0.0010 - acc: 1.0000 - val_loss: 0.0126 - val_acc: 1.0000
Epoch 63/100
4/4 [==============================] - 7s 2s/step - loss: 0.0026 - acc: 1.0000 - val_loss: 0.0154 - val_acc: 1.0000
Epoch 64/100
4/4 [==============================] - 6s 2s/step - loss: 0.0015 - acc: 1.0000 - val_loss: 0.0118 - val_acc: 1.0000
Epoch 65/100
4/4 [==============================] - 6s 2s/step - loss: 0.0032 - acc: 1.0000 - val_loss: 0.0172 - val_acc: 1.0000
Epoch 66/100
4/4 [==============================] - 6s 2s/step - loss: 0.0020 - acc: 1.0000 - val_loss: 0.0131 - val_acc: 1.0000
Epoch 67/100
4/4 [==============================] - 6s 2s/step - loss: 0.0037 - acc: 1.0000 - val_loss: 0.3394 - val_acc: 0.9333
Epoch 68/100
4/4 [==============================] - 6s 1s/step - loss: 0.0530 - acc: 0.9833 - val_loss: 0.0125 - val_acc: 1.0000
Epoch 69/100
4/4 [==============================] - 6s 2s/step - loss: 0.0016 - acc: 1.0000 - val_loss: 0.0110 - val_acc: 1.0000
Epoch 70/100
4/4 [==============================] - 6s 1s/step - loss: 0.0013 - acc: 1.0000 - val_loss: 0.0104 - val_acc: 1.0000
Epoch 71/100
4/4 [==============================] - 6s 2s/step - loss: 0.0204 - acc: 0.9921 - val_loss: 0.0097 - val_acc: 1.0000
Epoch 72/100
4/4 [==============================] - 4s 911ms/step - loss: 8.7974e-04 - acc: 1.0000 - val_loss: 0.0094 - val_acc: 1.0000
Epoch 73/100
4/4 [==============================] - 3s 709ms/step - loss: 5.8211e-04 - acc: 1.0000 - val_loss: 0.0105 - val_acc: 1.0000
Epoch 74/100
4/4 [==============================] - 5s 1s/step - loss: 8.9054e-04 - acc: 1.0000 - val_loss: 0.0088 - val_acc: 1.0000
Epoch 75/100
4/4 [==============================] - 6s 1s/step - loss: 4.3046e-04 - acc: 1.0000 - val_loss: 0.0113 - val_acc: 1.0000
Epoch 76/100
4/4 [==============================] - 6s 2s/step - loss: 0.0015 - acc: 1.0000 - val_loss: 0.0659 - val_acc: 0.9667
Epoch 77/100
4/4 [==============================] - 6s 2s/step - loss: 0.0876 - acc: 0.9745 - val_loss: 0.0082 - val_acc: 1.0000
Epoch 78/100
4/4 [==============================] - 6s 1s/step - loss: 0.0011 - acc: 1.0000 - val_loss: 0.0066 - val_acc: 1.0000
Epoch 79/100
4/4 [==============================] - 6s 1s/step - loss: 0.0013 - acc: 1.0000 - val_loss: 0.0053 - val_acc: 1.0000
Epoch 80/100
4/4 [==============================] - 6s 2s/step - loss: 8.9513e-04 - acc: 1.0000 - val_loss: 0.0050 - val_acc: 1.0000
Epoch 81/100
4/4 [==============================] - 6s 1s/step - loss: 9.1375e-04 - acc: 1.0000 - val_loss: 0.0050 - val_acc: 1.0000
Epoch 82/100
4/4 [==============================] - 6s 1s/step - loss: 9.5460e-04 - acc: 1.0000 - val_loss: 0.0044 - val_acc: 1.0000
Epoch 83/100
4/4 [==============================] - 6s 1s/step - loss: 6.8546e-04 - acc: 1.0000 - val_loss: 0.0043 - val_acc: 1.0000
Epoch 84/100
4/4 [==============================] - 6s 1s/step - loss: 9.3917e-04 - acc: 1.0000 - val_loss: 0.0040 - val_acc: 1.0000
Epoch 85/100
4/4 [==============================] - 6s 1s/step - loss: 3.2234e-04 - acc: 1.0000 - val_loss: 0.0038 - val_acc: 1.0000
Epoch 86/100
4/4 [==============================] - 6s 1s/step - loss: 0.0025 - acc: 1.0000 - val_loss: 0.0751 - val_acc: 0.9333
Epoch 87/100
4/4 [==============================] - 6s 2s/step - loss: 0.0018 - acc: 1.0000 - val_loss: 0.0267 - val_acc: 1.0000
Epoch 88/100
4/4 [==============================] - 6s 1s/step - loss: 0.0099 - acc: 1.0000 - val_loss: 0.3336 - val_acc: 0.9333
Epoch 89/100
4/4 [==============================] - 6s 1s/step - loss: 0.0501 - acc: 0.9921 - val_loss: 0.0032 - val_acc: 1.0000
Epoch 90/100
4/4 [==============================] - 6s 1s/step - loss: 0.0017 - acc: 1.0000 - val_loss: 0.0028 - val_acc: 1.0000
Epoch 91/100
4/4 [==============================] - 3s 862ms/step - loss: 0.0023 - acc: 1.0000 - val_loss: 0.0202 - val_acc: 1.0000
Epoch 92/100
4/4 [==============================] - 3s 704ms/step - loss: 0.0012 - acc: 1.0000 - val_loss: 0.0035 - val_acc: 1.0000
Epoch 93/100
4/4 [==============================] - 3s 749ms/step - loss: 0.0011 - acc: 1.0000 - val_loss: 0.0041 - val_acc: 1.0000
Epoch 94/100
4/4 [==============================] - 3s 708ms/step - loss: 8.7733e-04 - acc: 1.0000 - val_loss: 0.0042 - val_acc: 1.0000
Epoch 95/100
4/4 [==============================] - 3s 735ms/step - loss: 0.0022 - acc: 1.0000 - val_loss: 0.0150 - val_acc: 1.0000
Epoch 96/100
4/4 [==============================] - 3s 726ms/step - loss: 0.0427 - acc: 0.9754 - val_loss: 0.2064 - val_acc: 0.9333
Epoch 97/100
4/4 [==============================] - 3s 730ms/step - loss: 0.0076 - acc: 1.0000 - val_loss: 0.0017 - val_acc: 1.0000
Epoch 98/100
4/4 [==============================] - 3s 716ms/step - loss: 0.0010 - acc: 1.0000 - val_loss: 0.0018 - val_acc: 1.0000
Epoch 99/100
4/4 [==============================] - 3s 719ms/step - loss: 8.7921e-04 - acc: 1.0000 - val_loss: 0.0015 - val_acc: 1.0000
Epoch 100/100
4/4 [==============================] - 3s 686ms/step - loss: 4.4658e-04 - acc: 1.0000 - val_loss: 0.0016 - val_acc: 1.0000
In [18]:
model.save('fruits_classification_2.h5')

Visualizing intermediate activations

That is to dsplaying the feature maps that are output by various convolution and pooling layers in a network, given a certain input. It will visualize have 3 dimensions: width, height, and depth (channels). Each channel encodes relatively independent features. Before visualization, we should load the model above.

In [17]:
from keras.models import load_model

model = load_model('fruits_classification_2.h5')
model.summary()  # As a reminder.
Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_5 (Conv2D)            (None, 148, 148, 32)      896       
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 74, 74, 32)        0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 72, 72, 64)        18496     
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 36, 36, 64)        0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 34, 34, 128)       73856     
_________________________________________________________________
max_pooling2d_7 (MaxPooling2 (None, 17, 17, 128)       0         
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 15, 15, 128)       147584    
_________________________________________________________________
max_pooling2d_8 (MaxPooling2 (None, 7, 7, 128)         0         
_________________________________________________________________
flatten_2 (Flatten)          (None, 6272)              0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 6272)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 512)               3211776   
_________________________________________________________________
dense_4 (Dense)              (None, 1)                 513       
=================================================================
Total params: 3,453,121
Trainable params: 3,453,121
Non-trainable params: 0
_________________________________________________________________

Input a new image for testing:

In [18]:
img_path = 'orange_test1.jpg'

# We preprocess the image into a 4D tensor
from keras.preprocessing import image
import numpy as np

img = image.load_img(img_path, target_size=(150, 150))
img_tensor = image.img_to_array(img)
img_tensor = np.expand_dims(img_tensor, axis=0)
# Remember that the model was trained on inputs
# that were preprocessed in the following way:
img_tensor /= 255.

# Its shape is (1, 150, 150, 3)
print(img_tensor.shape)
(1, 150, 150, 3)

Display our picture:

In [19]:
import matplotlib.pyplot as plt

plt.imshow(img_tensor[0])
plt.show()

Create a Keras model that takes batches of images as input, and outputs the activations of all convolution and pooling layers. To do this, we will use the Keras class Model. This one has one input and 8 outputs, one output per layer activation:

In [20]:
from keras import models

# Extracts the outputs of the top 8 layers:
layer_outputs = [layer.output for layer in model.layers[:8]]
# Creates a model that will return these outputs, given the model input:
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
In [21]:
# This will return a list of 5 Numpy arrays:
# one array per layer activation
activations = activation_model.predict(img_tensor)
In [22]:
first_layer_activation = activations[0]
print(first_layer_activation.shape)
(1, 148, 148, 32)

It's a 148x148 feature map with 32 channels. Let's get the plot of some seperate channel

In [23]:
import matplotlib.pyplot as plt

plt.matshow(first_layer_activation[0, :, :, 3], cmap='viridis')
plt.show()

This is how the 3rd channel looks like

In [24]:
plt.matshow(first_layer_activation[0, :, :, 30], cmap='viridis')
plt.show()

This is how the 30th channel looks like. From this plot we can find the model can detect approximate outline for "orange"

Now plot a complete visualization of all the activations in the network:

In [25]:
import keras

# These are the names of the layers, so can have them as part of our plot
layer_names = []
for layer in model.layers[:8]:
    layer_names.append(layer.name)

images_per_row = 16

# Now let's display our feature maps
for layer_name, layer_activation in zip(layer_names, activations):
    # This is the number of features in the feature map
    n_features = layer_activation.shape[-1]

    # The feature map has shape (1, size, size, n_features)
    size = layer_activation.shape[1]

    # We will tile the activation channels in this matrix
    n_cols = n_features // images_per_row
    display_grid = np.zeros((size * n_cols, images_per_row * size))

    # We'll tile each filter into this big horizontal grid
    for col in range(n_cols):
        for row in range(images_per_row):
            channel_image = layer_activation[0,
                                             :, :,
                                             col * images_per_row + row]
            # Post-process the feature to make it visually palatable
            channel_image -= channel_image.mean()
            channel_image /= channel_image.std()
            channel_image *= 64
            channel_image += 128
            channel_image = np.clip(channel_image, 0, 255).astype('uint8')
            display_grid[col * size : (col + 1) * size,
                         row * size : (row + 1) * size] = channel_image

    # Display the grid
    scale = 1. / size
    plt.figure(figsize=(scale * display_grid.shape[1],
                        scale * display_grid.shape[0]))
    plt.title(layer_name)
    plt.grid(False)
    plt.imshow(display_grid, aspect='auto', cmap='viridis')
    
plt.show()
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:30: RuntimeWarning: invalid value encountered in true_divide

From the plot we can find that the lower layers can detect almost all the information for the initial image. As the layer goes higher-up, the activations become increasingly abstract and less visually interpretable. The higher layers can only remain some highlight part for part of edge of orange.

Visualizing convnet filters

That is to apply gradient descent to the value of the input image of a convnet so as to maximize the response of a specific filter, starting from a blank input image: build a loss function that maximizes the value of a given filter in a given convolution layer, then we will use stochastic gradient descent to adjust the values of the input image so as to maximize this activation value.

In [26]:
from keras.applications import VGG16
from keras import backend as K

model = VGG16(weights='imagenet',
              include_top=False)

layer_name = 'block3_conv1'
filter_index = 0

layer_output = model.get_layer(layer_name).output
loss = K.mean(layer_output[:, :, :, filter_index])

To implement gradient descent, we will need the gradient of this loss with respect to the model's input. To do this, we will use the gradients function packaged with the backend module of Keras:

In [27]:
# The call to `gradients` returns a list of tensors (of size 1 in this case)
# hence we only keep the first element -- which is a tensor.
grads = K.gradients(loss, model.input)[0]
In [28]:
# We add 1e-5 before dividing so as to avoid accidentally dividing by 0.
grads /= (K.sqrt(K.mean(K.square(grads))) + 1e-5)

Compute the value of the loss tensor and the gradient tensor. I will define a Keras backend function to do this: using iterate and returning a list of two Numpy tensors: the loss value and the gradient value.

In [29]:
iterate = K.function([model.input], [loss, grads])

# Let's test it:
import numpy as np
loss_value, grads_value = iterate([np.zeros((1, 150, 150, 3))])

Define a Python loop to do stochastic gradient descent:

In [30]:
# We start from a gray image with some noise
input_img_data = np.random.random((1, 150, 150, 3)) * 20 + 128.

# Run gradient ascent for 40 steps
step = 1.  # this is the magnitude of each gradient update
for i in range(40):
    # Compute the loss value and gradient value
    loss_value, grads_value = iterate([input_img_data])
    # Here we adjust the input image in the direction that maximizes the loss
    input_img_data += grads_value * step

Post-process this tensor to turn it into a displayable image using the straightforward utility function:

In [31]:
def deprocess_image(x):
    # normalize tensor: center on 0., ensure std is 0.1
    x -= x.mean()
    x /= (x.std() + 1e-5)
    x *= 0.1

    # clip to [0, 1]
    x += 0.5
    x = np.clip(x, 0, 1)

    # convert to RGB array
    x *= 255
    x = np.clip(x, 0, 255).astype('uint8')
    return x

Create a function that takes as input a layer name and a filter index, and that returns a valid image tensor representing the pattern that maximizes the activation the specified filter:

In [32]:
def generate_pattern(layer_name, filter_index, size=150):
    # Build a loss function that maximizes the activation
    # of the nth filter of the layer considered.
    layer_output = model.get_layer(layer_name).output
    loss = K.mean(layer_output[:, :, :, filter_index])

    # Compute the gradient of the input picture wrt this loss
    grads = K.gradients(loss, model.input)[0]

    # Normalization trick: we normalize the gradient
    grads /= (K.sqrt(K.mean(K.square(grads))) + 1e-5)

    # This function returns the loss and grads given the input picture
    iterate = K.function([model.input], [loss, grads])
    
    # We start from a gray image with some noise
    input_img_data = np.random.random((1, size, size, 3)) * 20 + 128.

    # Run gradient ascent for 40 steps
    step = 1.
    for i in range(40):
        loss_value, grads_value = iterate([input_img_data])
        input_img_data += grads_value * step
        
    img = input_img_data[0]
    return deprocess_image(img)

Use it to visualize filter 0 in layer block3_conv1:

In [33]:
plt.imshow(generate_pattern('block3_conv1', 0))
plt.show()

Visualize every single filter in every layer: but only look at the first 64 filters in each layer, and only look at the first layer of each convolution block (block1_conv1, block2_conv1, block3_conv1, block4_conv1, block5_conv1). That is a 8x8 grid of 64x64 filter patterns, with some black margins between each filter pattern.

In [34]:
for layer_name in ['block1_conv1', 'block2_conv1', 'block3_conv1', 'block4_conv1']:
    size = 64
    margin = 5

    # This a empty (black) image where we will store our results.
    results = np.zeros((8 * size + 7 * margin, 8 * size + 7 * margin, 3))

    for i in range(8):  # iterate over the rows of our results grid
        for j in range(8):  # iterate over the columns of our results grid
            # Generate the pattern for filter `i + (j * 8)` in `layer_name`
            filter_img = generate_pattern(layer_name, i + (j * 8), size=size)

            # Put the result in the square `(i, j)` of the results grid
            horizontal_start = i * size + i * margin
            horizontal_end = horizontal_start + size
            vertical_start = j * size + j * margin
            vertical_end = vertical_start + size
            results[horizontal_start: horizontal_end, vertical_start: vertical_end, :] = filter_img

    # Display the results grid
    plt.figure(figsize=(20, 20))
    plt.imshow((results * 255).astype(np.uint8))
    plt.show()

This is the filter visualizations: each layer in a convnet simply learns a collection of filters such that their inputs can be expressed as a combination of the filters. The filter in the lower layers only learn little thing from outside. As the layer goes higher-up, the filter can learn more information.

Visualizing heatmaps of class activation

That is to understand which parts of a given image led a convnet to its final classification decision. The techniques is called "Class Activation Map" (CAM) visualization, and consists in producing heatmaps of "class activation" over input images. To demonstrate this technique, I will also use the VGG16 network:

In [58]:
from keras.applications.vgg16 import VGG16

K.clear_session()

# Note that we are including the densely-connected classifier on top;
# all previous times, we were discarding it.
model = VGG16(weights='imagenet')

orange_test I also use the orange image using in visualizing intermediate activations

Load the image, resize it to 224x224, convert it to a Numpy float32 tensor, and apply these pre-processing rules.

In [59]:
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input, decode_predictions
import numpy as np

# The local path to our target image
img_path = 'orange_test1.jpg'

# `img` is a PIL image of size 224x224
img = image.load_img(img_path, target_size=(224, 224))

# `x` is a float32 Numpy array of shape (224, 224, 3)
x = image.img_to_array(img)

# We add a dimension to transform our array into a "batch"
# of size (1, 224, 224, 3)
x = np.expand_dims(x, axis=0)

# Finally we preprocess the batch
# (this does channel-wise color normalization)
x = preprocess_input(x)
In [60]:
preds = model.predict(x)
print('Predicted:', decode_predictions(preds, top=3)[0])
Predicted: [('n07747607', 'orange', 0.9933342), ('n07749582', 'lemon', 0.00665001), ('n07753592', 'banana', 6.735945e-06)]

From the result we can see that the VGG16 model predict it as orange(99.33%), lemon (0.67%), and banana (with very little probability)

In [61]:
np.argmax(preds[0])
Out[61]:
950

From the above part, the entry in the prediction vector that was maximally activated is the one corresponding to the "orange" class, at index 950 To visualize which parts of our image were the most "orange"-like, we should use the Grad-CAM process:

In [62]:
# This is the "african elephant" entry in the prediction vector
african_elephant_output = model.output[:, 950]

# The is the output feature map of the `block5_conv3` layer,
# the last convolutional layer in VGG16
last_conv_layer = model.get_layer('block5_conv3')

# This is the gradient of the "african elephant" class with regard to
# the output feature map of `block5_conv3`
grads = K.gradients(african_elephant_output, last_conv_layer.output)[0]

# This is a vector of shape (512,), where each entry
# is the mean intensity of the gradient over a specific feature map channel
pooled_grads = K.mean(grads, axis=(0, 1, 2))

# This function allows us to access the values of the quantities we just defined:
# `pooled_grads` and the output feature map of `block5_conv3`,
# given a sample image
iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])

# These are the values of these two quantities, as Numpy arrays,
# given our sample image of two elephants
pooled_grads_value, conv_layer_output_value = iterate([x])

# We multiply each channel in the feature map array
# by "how important this channel is" with regard to the elephant class
for i in range(512):
    conv_layer_output_value[:, :, i] *= pooled_grads_value[i]

# The channel-wise mean of the resulting feature map
# is our heatmap of class activation
heatmap = np.mean(conv_layer_output_value, axis=-1)

Normalize the heatmap between 0 and 1:

In [63]:
heatmap = np.maximum(heatmap, 0)
heatmap /= np.max(heatmap)
plt.matshow(heatmap)
plt.show()

Visualize which parts of our image were the most "orange"-like: this is plot is the most important part that makes the image recognized as an orange

In [64]:
!sudo apt-get update
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:3 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [766 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]      
Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]    
Get:6 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [4952 B]
Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [647 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [8385 B]
Get:10 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1284 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [19.0 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [946 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [7998 B]
Get:17 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4212 B]
Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [2496 B]
Fetched 17.1 MB in 8s (2135 kB/s)                                              
Reading package lists... Done
In [65]:
!sudo apt-get install -y libsm6 libxext6 libxrender-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libbsd0 libice6 libpthread-stubs0-dev libx11-6 libx11-data libx11-dev
  libx11-doc libxau-dev libxau6 libxcb1 libxcb1-dev libxdmcp-dev libxdmcp6
  libxrender1 multiarch-support x11-common x11proto-core-dev x11proto-dev
  xorg-sgml-doctools xtrans-dev
Suggested packages:
  libxcb-doc
The following NEW packages will be installed:
  libbsd0 libice6 libpthread-stubs0-dev libsm6 libx11-6 libx11-data libx11-dev
  libx11-doc libxau-dev libxau6 libxcb1 libxcb1-dev libxdmcp-dev libxdmcp6
  libxext6 libxrender-dev libxrender1 multiarch-support x11-common
  x11proto-core-dev x11proto-dev xorg-sgml-doctools xtrans-dev
0 upgraded, 23 newly installed, 0 to remove and 19 not upgraded.
Need to get 4109 kB of archives.
After this operation, 20.1 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 multiarch-support amd64 2.27-3ubuntu1 [6916 B]
Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxau6 amd64 1:1.0.8-1 [8376 B]
Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 libbsd0 amd64 0.8.7-1 [41.5 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxdmcp6 amd64 1:1.1.2-3 [10.7 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxcb1 amd64 1.13-2~ubuntu18.04 [45.5 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libx11-data all 2:1.6.4-3ubuntu0.2 [113 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libx11-6 amd64 2:1.6.4-3ubuntu0.2 [569 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxext6 amd64 2:1.3.3-1 [29.4 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 x11-common all 1:7.7+19ubuntu7.1 [22.5 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libice6 amd64 2:1.0.9-2 [40.2 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsm6 amd64 2:1.2.2-1 [15.8 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpthread-stubs0-dev amd64 0.3-4 [4068 B]
Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 xorg-sgml-doctools all 1:1.11-1 [12.9 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-dev all 2018.4-4 [251 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11proto-core-dev all 2018.4-4 [2620 B]
Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxau-dev amd64 1:1.0.8-1 [11.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxdmcp-dev amd64 1:1.1.2-3 [25.1 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 xtrans-dev all 1.3.5-1 [70.5 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxcb1-dev amd64 1.13-2~ubuntu18.04 [80.0 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libx11-dev amd64 2:1.6.4-3ubuntu0.2 [640 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libx11-doc all 2:1.6.4-3ubuntu0.2 [2065 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxrender1 amd64 1:0.9.10-1 [18.7 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxrender-dev amd64 1:0.9.10-1 [24.9 kB]
Fetched 4109 kB in 2s (2015 kB/s)     
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package multiarch-support.
(Reading database ... 15578 files and directories currently installed.)
Preparing to unpack .../multiarch-support_2.27-3ubuntu1_amd64.deb ...
Unpacking multiarch-support (2.27-3ubuntu1) ...
Setting up multiarch-support (2.27-3ubuntu1) ...
Selecting previously unselected package libxau6:amd64.
(Reading database ... 15581 files and directories currently installed.)
Preparing to unpack .../00-libxau6_1%3a1.0.8-1_amd64.deb ...
Unpacking libxau6:amd64 (1:1.0.8-1) ...
Selecting previously unselected package libbsd0:amd64.
Preparing to unpack .../01-libbsd0_0.8.7-1_amd64.deb ...
Unpacking libbsd0:amd64 (0.8.7-1) ...
Selecting previously unselected package libxdmcp6:amd64.
Preparing to unpack .../02-libxdmcp6_1%3a1.1.2-3_amd64.deb ...
Unpacking libxdmcp6:amd64 (1:1.1.2-3) ...
Selecting previously unselected package libxcb1:amd64.
Preparing to unpack .../03-libxcb1_1.13-2~ubuntu18.04_amd64.deb ...
Unpacking libxcb1:amd64 (1.13-2~ubuntu18.04) ...
Selecting previously unselected package libx11-data.
Preparing to unpack .../04-libx11-data_2%3a1.6.4-3ubuntu0.2_all.deb ...
Unpacking libx11-data (2:1.6.4-3ubuntu0.2) ...
Selecting previously unselected package libx11-6:amd64.
Preparing to unpack .../05-libx11-6_2%3a1.6.4-3ubuntu0.2_amd64.deb ...
Unpacking libx11-6:amd64 (2:1.6.4-3ubuntu0.2) ...
Selecting previously unselected package libxext6:amd64.
Preparing to unpack .../06-libxext6_2%3a1.3.3-1_amd64.deb ...
Unpacking libxext6:amd64 (2:1.3.3-1) ...
Selecting previously unselected package x11-common.
Preparing to unpack .../07-x11-common_1%3a7.7+19ubuntu7.1_all.deb ...
dpkg-query: no packages found matching nux-tools
Unpacking x11-common (1:7.7+19ubuntu7.1) ...
Selecting previously unselected package libice6:amd64.
Preparing to unpack .../08-libice6_2%3a1.0.9-2_amd64.deb ...
Unpacking libice6:amd64 (2:1.0.9-2) ...
Selecting previously unselected package libsm6:amd64.
Preparing to unpack .../09-libsm6_2%3a1.2.2-1_amd64.deb ...
Unpacking libsm6:amd64 (2:1.2.2-1) ...
Selecting previously unselected package libpthread-stubs0-dev:amd64.
Preparing to unpack .../10-libpthread-stubs0-dev_0.3-4_amd64.deb ...
Unpacking libpthread-stubs0-dev:amd64 (0.3-4) ...
Selecting previously unselected package xorg-sgml-doctools.
Preparing to unpack .../11-xorg-sgml-doctools_1%3a1.11-1_all.deb ...
Unpacking xorg-sgml-doctools (1:1.11-1) ...
Selecting previously unselected package x11proto-dev.
Preparing to unpack .../12-x11proto-dev_2018.4-4_all.deb ...
Unpacking x11proto-dev (2018.4-4) ...
Selecting previously unselected package x11proto-core-dev.
Preparing to unpack .../13-x11proto-core-dev_2018.4-4_all.deb ...
Unpacking x11proto-core-dev (2018.4-4) ...
Selecting previously unselected package libxau-dev:amd64.
Preparing to unpack .../14-libxau-dev_1%3a1.0.8-1_amd64.deb ...
Unpacking libxau-dev:amd64 (1:1.0.8-1) ...
Selecting previously unselected package libxdmcp-dev:amd64.
Preparing to unpack .../15-libxdmcp-dev_1%3a1.1.2-3_amd64.deb ...
Unpacking libxdmcp-dev:amd64 (1:1.1.2-3) ...
Selecting previously unselected package xtrans-dev.
Preparing to unpack .../16-xtrans-dev_1.3.5-1_all.deb ...
Unpacking xtrans-dev (1.3.5-1) ...
Selecting previously unselected package libxcb1-dev:amd64.
Preparing to unpack .../17-libxcb1-dev_1.13-2~ubuntu18.04_amd64.deb ...
Unpacking libxcb1-dev:amd64 (1.13-2~ubuntu18.04) ...
Selecting previously unselected package libx11-dev:amd64.
Preparing to unpack .../18-libx11-dev_2%3a1.6.4-3ubuntu0.2_amd64.deb ...
Unpacking libx11-dev:amd64 (2:1.6.4-3ubuntu0.2) ...
Selecting previously unselected package libx11-doc.
Preparing to unpack .../19-libx11-doc_2%3a1.6.4-3ubuntu0.2_all.deb ...
Unpacking libx11-doc (2:1.6.4-3ubuntu0.2) ...
Selecting previously unselected package libxrender1:amd64.
Preparing to unpack .../20-libxrender1_1%3a0.9.10-1_amd64.deb ...
Unpacking libxrender1:amd64 (1:0.9.10-1) ...
Selecting previously unselected package libxrender-dev:amd64.
Preparing to unpack .../21-libxrender-dev_1%3a0.9.10-1_amd64.deb ...
Unpacking libxrender-dev:amd64 (1:0.9.10-1) ...
Setting up libpthread-stubs0-dev:amd64 (0.3-4) ...
Setting up xorg-sgml-doctools (1:1.11-1) ...
Setting up libbsd0:amd64 (0.8.7-1) ...
Setting up x11proto-dev (2018.4-4) ...
Setting up xtrans-dev (1.3.5-1) ...
Setting up libx11-doc (2:1.6.4-3ubuntu0.2) ...
Setting up libxdmcp6:amd64 (1:1.1.2-3) ...
Setting up x11-common (1:7.7+19ubuntu7.1) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up libx11-data (2:1.6.4-3ubuntu0.2) ...
Setting up libxau6:amd64 (1:1.0.8-1) ...
Setting up x11proto-core-dev (2018.4-4) ...
Setting up libxau-dev:amd64 (1:1.0.8-1) ...
Setting up libxdmcp-dev:amd64 (1:1.1.2-3) ...
Setting up libice6:amd64 (2:1.0.9-2) ...
Setting up libxcb1:amd64 (1.13-2~ubuntu18.04) ...
Setting up libsm6:amd64 (2:1.2.2-1) ...
Setting up libx11-6:amd64 (2:1.6.4-3ubuntu0.2) ...
Setting up libxrender1:amd64 (1:0.9.10-1) ...
Setting up libxcb1-dev:amd64 (1.13-2~ubuntu18.04) ...
Setting up libx11-dev:amd64 (2:1.6.4-3ubuntu0.2) ...
Setting up libxext6:amd64 (2:1.3.3-1) ...
Setting up libxrender-dev:amd64 (1:0.9.10-1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
In [66]:
!sudo pip install opencv-python
WARNING: The directory '/home/jovyan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/home/jovyan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting opencv-python
  Downloading https://files.pythonhosted.org/packages/5e/7e/bd5425f4dacb73367fddc71388a47c1ea570839197c2bcad86478e565186/opencv_python-4.1.1.26-cp36-cp36m-manylinux1_x86_64.whl (28.7MB)
     |████████████████████████████████| 28.7MB 3.0MB/s eta 0:00:01
Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python) (1.16.4)
Installing collected packages: opencv-python
Successfully installed opencv-python-4.1.1.26
WARNING: You are using pip version 19.1.1, however version 19.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Use OpenCV to generate an image that superimposes the original image with the heatmap:

In [68]:
import cv2

# We use cv2 to load the original image
img = cv2.imread(img_path)

# We resize the heatmap to have the same size as the original image
heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))

# We convert the heatmap to RGB
heatmap = np.uint8(255 * heatmap)

# We apply the heatmap to the original image
heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)

# 0.4 here is a heatmap intensity factor
superimposed_img = heatmap * 0.4 + img

# Save the image to disk
cv2.imwrite('my_orangetest.jpg', superimposed_img)
Out[68]:
True

Orange Test Result

From the result heatmap we can see that the outside color and the outline are very import for the model to recognize an orange